A System-Theoretical View on Local Motion Estimation (invited)

The problem of estimating local motion in the presence of noise deserves a thorough system-theoretical analysis. In current textbooks, the predominant procedures are classified into differential (’optical flow’) approaches and matching (or correlation) methods. From a system-theoretical point of view, both classes can be interpreted as a set of linear filters followed by a simple nonlinear operation. However, neither the definition of motion in terms of spatiotemporal gradients nor the minimization of loss functions on two subsequent image patches captures the full essence of what motion means, i.e. the preference direction of a spatiotemporal signal of reduced intrinsic dimensionality embedded in noise. We will discuss the elements that influence the possible precision of local motion estimation methods, and describe estimation-theoretic approaches which subsume tradition methods, on one hand, and give perspectives for signal-dependent optimization, on the other hand. 1. Models for oriented signals and spatiotemporal flow Let us regard a three-dimensional space-time volume spanned by the coordinates (x;y; t) (x1;x2;x3) x; x 2 IR3 on which the signal s(x) is defined. A local neighborhood with ideal orientation can be described as s(x) = f2(n1 x; n T 2 x) ; (1) where f2( ; ) is a scalar function of two arguments, and n1;n2 are two orthonormal vectors perpendicular to the local motion vector r.We denote each signal s(x) for which it is possible to find a coordinate frame n1;n2;r according to eq.1 (i.e. such that the signal varies only in 2 directions), as a rank 2 signal in a three-dimensional space [7]. We obtain such signals if a two-dimensional image which shows gray value variations in both spatial directions is translated linearly with constant speed. For a more detailed analysis of orientation, it makes sense to describe the class of ’allowed’ two-dimensional image signals which are implicitly used in eq. 1, for instance a) by specifying a functional model (bandlimited, splineinterpolatable, ...), b) or by interpreting the signal as a sample from a suitable stochastic process (which means that basically all realizations are allowed, but with possibly very different measures of probability). Once defined, such a model for the 2D image, together with the motion vector r as a parameter, defines a specific class Co of oriented signals s(x). I stress that other models are imaginable and have been used successfully, such as models incorporating affine transformations instead of pure translation (for an overview cf. [9]), or even including complex spatio-temporal variation of the image signal (e.g. diffusion) [8]. However, for the purpose of a compact presentation of some central concepts, these models are not further discussed here. 2. Differential approaches The general principle behind all differential approaches is that the conservation of some local image characteristic throughout its temporal evolution is reflected in terms of differential-geometric descriptors. In its simplest form, the assumed conservation of brightness along the motion trajectory through space–time leads to the well-known brightness constancy constraint equation (BCCE), where g(x) is the gradient of the gray value signal s(x): ∂s ∂x1 ; ∂s ∂x2 ; ∂s ∂x3 r = 0 , g (x) r = 0: (2) Since gT (x) r is proportional to the directional derivative of s in direction r, the BCCE states that this derivative vanishes in the direction of motion. In order to cope with the aperture problem in case that the signal is locally only a rank 1 signal, and in order to decrease the variance of the motion vector estimate, usually some kind of weighted averaging is performed in a neighborhood V , using a weight function w(x). This leads to the optimization problem Z V w(x) g (x) r 2 dx ! min (3)